How Cognitive Testing Improves Market Research

In the fast-paced industry of market research, the pressure to accelerate processes is ever-present. One may contemplate bypassing certain steps to expedite data collection. However, in doing so, you risk compromising the integrity of your insights. For instance, what if you later down the line discover that respondents misinterpreted your survey questions? This will create skewed data and could potentially lead to decision-making that’s not based on actual customer insight.

To illustrate, imagine you are a coffee bean producer and you are sending out a survey to learn more about people’s coffee consumption. In your survey, you ask the question “How many cups of coffee do you drink per day?” Now this question seems simple enough to answer, however, it actually introduces ambiguity in at least two aspects: the definition of a "cup" and the type of "coffee".

The term "cup" can be interpreted differently; some may think of a standard 6 oz cup, while others may consider a larger or smaller serving. Additionally, "coffee" can include various types such as brewed coffee, espresso, or specialty drinks, each having different caffeine content and serving sizes. As a result, respondents may provide varied and potentially inaccurate data due to the lack of specificity in defining the unit and the beverage. Ultimately, the information you will get from this question tells you very little about how much ground coffee people consume. 

So how do you go about bringing clarity to this ambiguity? Conduct a cognitive test with your survey! This indispensable tool in the market researcher’s toolkit acts as a safeguard against misunderstanding and ensures the delivery of superior data quality.

‍

How To Conduct Cognitive Testing

To facilitate cognitive testing, an interviewer can administer the survey questions to respondents and encourage them to share the thought processes they go through when answering specific questions. Alternatively, the interviewer can observe the respondents clicking through the survey themselves and ask probing questions. Some of the techniques they may use include:

‍Thinking aloud
The interviewer asks respondents to provide free-form, stream-of-consciousness style answers about what is going through their minds as they answer the questions, for example:

• Walk me through your thought processes as you answer this question
• As you answer, tell me what’s going through your mind

This method is especially helpful if you want insight into how respondents recall information, which can potentially reveal related concepts worth exploring further. 

‍Active probing
The interviewer asks planned (proactive) or spontaneous (reactive) questions to delve into specific points or follow up on respondents’ remarks. The former approach poses prewritten questions to all respondents to explore a certain research question, for example:  

• What does that word or phrase mean to you in the context of this question?
• How did you come up with your answer?
• Can you explain what this question is trying to get at in your own words?

Whereas reactive probing occurs when the interviewer wants to further explore respondents’ unpredictable answers to reveal insights the script doesn’t account for, for example:

• Why did you mention X (unexpected term)? 
• Can you tell me more about your answer? 

Interviewing a variety of people helps uncover a wide range of perspectives and potential pitfalls. Research shows that as your sample size increases, additional problems continue to be detected (Blair & Conrad, 2011). As a rule of thumb, we recommend conducting at least 10-15 interviews or more if the survey is very complex and research constraints allow. 

For multinational research, it can also be very helpful to recruit participants from various markets to uncover regional differences in how the subject matter is understood. For example, in a past study we presented property rental types in a list to different countries, and through cognitive testing we discovered that respondents from Brazil didn’t recognize condos or condominiums as a common rental term. Backed with this insight, we could amend the survey and hide this item from the list of options for Brazilian respondents. 

‍

What Are The Benefits of Cognitive Testing?

One can think of cognitive testing as a sort of “insurance” for market research. By that, we mean that it helps to ensure that the data we collect reflects how respondents really think and feel and is accurate. By doing cognitive testing we can:

• Ensure that respondents understand the question as it was intended
• Confirm that respondents can recall the appropriate information required to answer questions
• Ensure that respondents can find their answers in the provided item lists or answer categories
• Discover unexpected pitfalls in areas that were not considered problematic before

For clients, this means you gain an improved survey design that works as intended and captures high-quality data. As a bonus, it also allows you to ask some qualitative interview questions which can add more texture to your quantitative data. 

To sum up, cognitive testing is a crucial step in market research and the importance thereof should never be underestimated. To make informed, data-driven decisions, you should ensure that your data must first and foremost be accurate and truly reflect the feelings and thoughts of your customer base. If you want to improve your market research with cognitive testing contact us.

‍

Sources:

‍Blair, J. and Conrad, F.G. (2011) ‘Sample size for cognitive interview pretesting’, Public Opinion Quarterly, 75(4), pp. 636–658. doi:10.1093/poq/nfr035.